Make build-time test failures non-fatal if 3 out of 5 attempts succeed

There are several upstream bugs that cause intermittent test failures,
and can intermittently be reproduced in real use. However, these are not
regressions, so we should not FTBFS just because we happen to have been
unlucky during build.

diff --git a/debian/changelog b/debian/changelog
index 61e7795ad79e67aecbcb584bfe4cc6f5b68b36b8..3b03577299193d5f4d58969c49b7a63f1914ce17 100644 (file)
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,15 @@
+ostree (2016.14-2) UNRELEASED; urgency=medium
+
+  * Make build-time test failures non-fatal, as long as at least
+    3 out of 5 attempts succeed.
+
+    There are several upstream bugs that cause intermittent test
+    failures, and can intermittently be reproduced in real use.
+    However, these are not regressions, so we should not FTBFS just
+    because we happen to have been unlucky during build.
+
+ -- Simon McVittie <smcv@debian.org>  Thu, 01 Dec 2016 12:29:00 +0000
+
 ostree (2016.14-1) unstable; urgency=medium
 
   * Switch the build-dependency on libgpgme11-dev (which no longer exists

diff --git a/debian/test.sh b/debian/test.sh
index dc8eed282de01988e3e77f11bf42f1a133b29bc8..caeaf1698b9d7a8cac615a6b9cdd7950a3fbcadb 100755 (executable)
--- a/debian/test.sh
+++ b/debian/test.sh
@@ -34,13 +34,18 @@ if pgrep lt-ostree || pgrep --full "gpg-agent --homedir /var/tmp/tap-test."; the
     pgrep lt-ostree | xargs --no-run-if-empty ps ww
 fi
 
-if [ "$failed" -gt 0 ]; then
+# There are several race conditions that cause intermittent failures.
+# They are not actually a regression - we've just been luckier in the
+# past - so let newer versions build reliably.
+if [ "$failed" -gt 2 ]; then
     echo "Failed $failed out of $try_tests test runs"
     if [ -z "$ignore" ]; then
         exit 1
     else
         echo "Ignoring test failure for this architecture"
     fi
+elif [ "$failed" -gt 0 ]; then
+    echo "Failed $failed out of $try_tests test runs; continuing anyway"
 fi
 
 exit 0